What is the Quality of Architectural Design Quality Metrics?

As software ages, its quality degrades, leading to pathologies such as architectural decay. Architectural decay is hard to fight against. First, it is a slow drifting process that results from the natural evolution of software as new features pull the system in new directions. It is not something that one can fight against on a day to day basis. Second, restructuring a legacy system is a costly and dangerous operation that has the potential to affect the entire code base. A daring prospect on systems of multi-million lines of code. This spawned a lot of research such as quality metrics to assess an architecture, or approaches to propose better architectures for legacy systems. Yet the architectural design quality metrics used were not adequately tested nor proved. We are therefore left to measure an unknown quality with unknown instruments. One of the reasons for this seems to be that building a formal experiment to test these issues is difficult due to the fuzzy nature of what is a good architecture, and to the need to use real big systems. In this paper we setup an experimental design with well know systems such as Eclipse or JHotDraw that would allow to test the relevance of architectural design metrics. We illustrate it on some well known metrics.